Hyperfast second-order local solvers for efficient statistically preconditioned distributed optimization

Statistical preconditioning enables fast methods for distributed large-scale empirical risk minimization problems. In this approach, multiple worker nodes compute gradients in parallel, which are then used by the central node to update parameter solving an auxiliary (preconditioned) smaller-scale optimization problem. The recently proposed Statistically Preconditioned Accelerated Gradient (SPAG) method [1] has complexity bounds superior other such algorithms but requires exact solution computationally intensive problems at every iteration. paper, we propose Inexact SPAG (InSPAG) and explicitly characterize accuracy corresponding subproblem needs be solved guarantee same convergence rate as method. We build our results first developing inexact adaptive accelerated Bregman proximal gradient general under relative smoothness strong convexity assumptions, may of independent interest. Moreover, explore properties problem InSPAG algorithm assuming Lipschitz third-order derivatives strongly convexity. For class, develop a linearly convergent Hyperfast second-order estimate total with hyperfast solver. Finally, illustrate method's practical efficiency performing numerical experiments on logistic regression models. To best knowledge, these implementing high-order problems, work data where dimension is order 3 million, number samples 700 million.